An Energy-Based Recurrent Neural Network for Multiple Fundamental Frequency Estimation

Many naturally occurring phenomena such as music, speech, or human motion are inherently sequential. Complex sequences are often non-local (longterm temporal dependencies) and high-dimensional (multi-modal conditional distribution). For the example of polyphonic music, these properties represent the basic components of Western music, namely rhythm and harmony. Here we wish to exploit the recurrent neural network (RNN) internal memory that can in principle represent long-term dependencies, and energy-based models, such as the Restricted Boltzmann Machine (RBM), that allow us to express complex distributions by the means of an energy function. This combination was first put forward with the so-called Temporal RBM (TRBM) [3], the first such probabilistic model which however uses a heuristic training procedure. The Recurrent TRBM (RTRBM) [4] is a slight modification of the TRBM that allows for exact inference and efficient training by contrastive divergence (CD). The RTRBM can be understood as a sequence of RBMs whose parameters